Description

Users were experiencing kurtosis service logs taking a long time. After running tests, I discovered that a majority of execution time during log processing was spent in the following lines:

logLines := []logline.LogLine{*logLine}
userServicesLogLinesMap := map[service.ServiceUUID][]logline.LogLine{
      serviceUuid: logLines,
}
logsByKurtosisUserServiceUuidChan <- userServicesLogLinesMap

Prior to this change, we were sending logs one at a time on an unbuffered channel - unbuffered channels block until the receiving goroutine reads the value. This was causing a lot of time being wasted waiting to send log lines across the channel.

This change implements a LogLineSender that:

1. uses a buffered go channel (won't block on sending line unless buffer is full)
2. batches log lines (reduces read overhead, receiving goroutine performs fewer reads/sends)

With this change, the time to read 20 minutes of cl-lighthouse-geth logs with log level set to debug went from 1min53sec to 30.055 seconds. The time to read 2 hours 10 minutes worth of cl-lighthouse debug logs (around 3.4 gb of logs) went from 15min1sec to 3min31 sec. (As a benchmark, cat logs.json on 3.4 gb of logs takes around 2min - on my machine - so much closer) This can likely be improved further by tuning the buffer size and batch amount.

Is this change user facing?

YES

References:

https://discord.com/channels/783719264308953108/1267837033032974467/1267842228072611881